Magma: A Ground-Truth Fuzzing Benchmark

High scalability and low running costs have made fuzz testing the de facto standard for discovering software bugs. Fuzzing techniques are constantly being improved in a race to build ultimate bug-finding tool. However, while fuzzing excels at finding bugs wild, evaluating comparing fuzzer performance is challenging due lack of metrics benchmarks. For example, crash count---perhaps most commonly-used metric---is inaccurate imperfections deduplication techniques. Additionally, unified set targets results ad hoc evaluations that hinder fair comparison. We tackle these problems by developing Magma, ground-truth benchmark enables uniform evaluation By introducing real into software, Magma allows realistic fuzzers against broad targets. instrumenting bugs, also collection bug-centric independent fuzzer. an open consisting seven perform variety input manipulations complex computations, presenting challenge state-of-the-art fuzzers. evaluate widely-used mutation-based (AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, SymCC-AFL) over 200,000 CPU-hours. Based on number reached, triggered, detected, we draw conclusions about fuzzers' exploration detection capabilities. This provides insight evaluation, highlighting importance ground truth performing more accurate meaningful evaluations.